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Using an on-line tool to investigate chemical engineering seniors’ 
concept of the design process 

Ruth A. Streveler 1 , Ronald L. Miller, Thomas M. Boyd 
Colorado School of Mines 



Abstract 

Engineers must understand how to design products and processes, and therefore design 
courses are often the culminating experience in an engineering student’s undergraduate 
education. How well do graduating seniors understand the concept of “design?” In this 
study, multidimensional scaling (MDS) was used to measure how chemical engineering 
seniors categorized key design terms at the beginning and end of a capstone design 
course. An on-line method was developed to collect the MDS data. The results suggest 
that some important design concepts were not well understood, even at the end of the 
semester. The instructional implications of these results are discussed. 

Introduction 

The design process is a cornerstone of engineering and is a common subject of 
engineering education research (Atman & Bursic, 1998; Wankat, 1999). Since practicing 
engineers must understand how to design products and processes, engineering students 
often are required to complete a “design” course in their senior year. These courses are 
often called “capstone” courses, because they are seen as the culminating experience in 
an engineering student’s undergraduate education. 

How do students on the threshold of entering the chemical engineering profession view 
this key concept? In this study we have investigated how chemical engineering seniors in 
a capstone design course view the concept of “design.” We also discuss how our results 
may provide feedback to engineering educators to improve instruction. 

In this study, we use multidimensional scaling (MDS) to measure cognitive structure (or 
the underlying configuration of a person’s knowledge). Multidimensional scaling 
(Kruskal & Wish, 1978) is the name for a family of methods that convert similarity 
judgments into physical distance. The use of MDS for representing cognitive structure 
has been validated by Champagne, Hoz & Klopfer (1984) and Shavelson (1985) and has 
been used in this way by Markham, Mintzes, & Jones (1994) and Streveler (1994). 

The first step in MDS analysis is to ask participants to group together terms that they 
believe to be conceptually linked. This information is averaged and converted to a 
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similarity matrix that is then input into the MDS procedure for analysis. MDS analysis 
then finds a model (in 2 to n dimensions) that is the best fit of the similarity matrix. 

In the MDS results, items seen as similar to one another will be plotted closely together. 
One can look at the MDS output, see which items students view as belonging together, 
and gain insight into how students are organizing the information they are learning. If the 
MDS analysis reveals terms clustered together “incorrectly” this may signify a potential 
misconception. A “misplaced” term, in other words one that is placed with other terms in 
an unexpected way, may be an indicator of a misconception about that term (Streveler & 
Bail, 1992; Streveler & Miller, 1999). 

We suggest that using MDS to help locate areas of student misunderstanding may be seen 
as a useful classroom assessment technique. The purpose of classroom assessment 
techniques is to give instructors informal feedback about what their students are learning 
(Angelo & Cross, 1993; Cross & Steadman, 1996) and the results of classroom 
assessment techniques are best used to improve learning (Cross, 1998). As will be seen 
in this paper, the MDS results served this purpose in this study. We also created an on- 
line method for gathering MDS data from students that has the potential to make this 
rather esoteric form of analysis more accessible to classroom instructors. 

Methods 

Participants in our study were 23 chemical engineering seniors enrolled in a capstone 
senior design course at the Colorado School of Mines. As stated above, we used MDS to 
measure the students’ cognitive structure. The following steps were used to conduct this 
study. 

1) The course professor selected terms that were deemed to be central to 
understanding chemical engineering design. The 32 terms selected by the 
instructor for the clustering task are listed in Table 1. 

2) In order to increase the ease of use, a method of gathering the clustering data on- 
line was developed and used in this study. As shown in Figure 1, a web page was 
created which listed the 32 terms in alphabetical order. Radio buttons allowed 
students to cluster the terms into one of nine possible groups. A PERL database 
was created to collect the data. The database collected the name of the student, 
their id number, the date the clustering task was completed, and the group number 
of each of the 32 terms. 

3) At the beginning of the semester, students were given the extra credit assignment 
of going to the class web page, and clustering the 32 terms into logical groups. A 
total of 22 students completed the pretest (96% of students registered for the 
design course). 
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Table 1. Design terms, selected by the instructor and used by students 
in the clustering task. 



Term number 


Term name 


Term 1 


capital cost 


Term 2 


cash flow analysis 


Term 3 


conceptual design 


Term 4 


debottlenecking 


Term 5 


design heuristic 


Term 6 


economic optimum 


Term 7 


energy transfer block 


Term 8 


engineering design process 


Term 9 


engineering judgment 


Term 10 


equipment heuristic 


Term 1 1 


generic block flow diagram 


Term 12 


HAZOP analysis 


Term 13 


input-output diagram 


Term 14 


life-cycle analysis 


Term 15 


maintenance cost 


Term 16 


operating cost 


Term 17 


operating heuristic 


Term 18 


piping and instrumentation diagram 


Term 19 


process analysis 


Term 20 


process bottleneck 


Term 21 


process evaluation 


Term 22 


process flow diagram 


Term 23 


process optimization 


Term 24 


process simulation 


Term 25 


process synthesis 


Term 26 


rate of return 


Term 27 


reaction block 


Term 28 


risk analysis 


Term 29 


separation block 


Term 30 


technical optimum 


Term 31 


time value of money 


Term 32 


troubleshooting 
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Figure 1 . Screen Capture of the MDS Webpage 



4) The assignment outlined in step 3 was repeated during the last week of 
instruction. A total of 14 students completed the posttest (61% of the students in 
the course). 

5) The database described in step 2 was emailed to the investigators after the 
students had completed the pretest, and again after the posttest was completed. 

6) These data were then input into the statistical package, SAS, and a similarity 
matrix was computed. This matrix was then used as the input to compute the 
MDS solution. 



Results 

The model representing the MDS solution may be expressed in 2 to n dimensions. A 
statistic called “stress” gives us a goodness-of-fit measure. The “Stress” measure for a 
two-dimensional solution was below 1.5 on both the pretest and posttest. This result 
showed that a 2-dimesional solution was adequate in both cases. 
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The two-dimensional solutions for the pretest (Figure 2) and posttest (Figure 3) are 
shown below. It is customary to name the axis dimensions expressed in a MDS solution 
(much as one names factors in factor analysis). Based on the clustering results obtained, 
we labeled the x-axis “economic vs. technical” and the y-axis “process vs. product.” 
(Refer to Table 1 for the listing of term numbers and term names.) The solution of the 
pretest was rotated for more consistent placement of pretest and posttest clusters in the 
same quadrants, thus allowing for easier pretest-posttest comparisons. 




Figure 2. Pretest Results of MDS Analysis for Senior Design Terms 
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Figure 3. Posttest Results of MDS Analysis for Senior Design Terms 



In the pretest (Figure 2) four distinct clusters of terms were observed: 

1- A cluster containing all terms that pertained to the economic analysis of a project: 
capital cost (Term 1), cash flow analysis (Term 2), economic optimum (Term 6), 
maintenance cost (Term 15), operating cost (Term 16), rate of return (Term 26), time 
value of money (Term 3 1). We will call this grouping Cluster 1 . These terms are 
shown as circles in Figures 2 and 3. 

2- A cluster listing process flow diagram terms: energy transfer block (Term 7), generic 
block flow diagram (Term 1 1), input-output diagram (Term 13), piping and 
instrumentation diagram (Term 18), process flow diagram (Term 22), reaction block 
(Term 27), separation block (Term 29). We will call this grouping Cluster 2. These 
terms are shown as squares in Figures 2 and 3. 

3- A cluster containing terms that described analysis of processes: debottlenecking 
(Term 4), process analysis (Term 19), process bottleneck (Term 20), process 
evaluation (Term 21), process optimization (Term 23), process simulation (Term 24), 
process synthesis (Term 25). We will call this grouping Cluster 3. These terms are 
shown as X’s in Figures 2 and 3. 



O 
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4- And a cluster with three terms containing the word heuristic: design heuristic (Term 
5), equipment heuristic (Term 10), operating heuristic (Terml7). We will call this 
grouping Cluster 4. These terms are shown as triangles in Figures 2 and 3. 

Other terms were scattered around the plot and could not easily be placed into clusters in 
the pretest. These terms are: conceptual design (Term 3), engineering design process 
(Term 8), engineering judgment (Term 9), HAZOP analysis (Term 12), life-cycle 
analysis (Term 14), risk analysis (Term 28), technical optimum (Term 30) and 
troubleshooting (Term 32). These terms are shown as diamonds in Figures 2 and 3. 

How does the position of these terms change in the posttest? As shown in Figure 3, 
Clusters 1, 2, 3, and 4 all remain in the posttest. However, some of terms are added to 
Cluster 3 (analysis of processes) in the posttest. The terms conceptual design (Term 3), 
engineering design process (Term 8), and engineering judgment (Terms 9), HAZOP 
analysis (Term 12) and technical optimum (Term 30) all become part of Cluster 3. 

The three heuristic terms (Terms 5, 10, and 17) in Cluster 4, become so closely clustered 
in the posttest that they merge into a single point. However, these three points also 
become more closely associated with Cluster 3. 

Three terms: life-cycle analysis (Term 14), risk analysis (Term 28), and troubleshooting 
(Term 32) continued to not be closely associated with any cluster of terms in the posttest. 

Discussion 

The design instructor determined that Clusters 1 and 2 were clustered accurately in both 
the pretest and posttest and we can infer that students have adequate understanding of the 
terms in these areas. Students included all terms dealing with economic analysis in 
Cluster 1 and indeed students in this capstone design course have previously completed 
an engineering economics course and did have a strong background in economics. 
Students also are familiar with design terms and this is shown in the clustering of terms in 
Cluster 2. 

In the pretest, Cluster 3 contained many terms which include the word “process;” 
however inclusion of the word “debottlenecking” in this cluster suggests that students 
used deeper characteristics to cluster these terms. 

In the posttest, five terms were added to Cluster 3. The terms conceptual design (Term 3), 
engineering design process (Term 8), and engineering judgment (Terms 9) now become 
part of Cluster 3 suggesting that students learned the importance of conceptualizing 
processes using engineering judgment when designing and analyzing chemical processes. 
The terms HAZOP analysis (Term 12) and technical optimum (Term 30) also became 
part of Cluster 3 in the posttest, which may indicate that students in the course learned 
about the importance of analyzing a process for the potential of hazardous operations 
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(HAZOP) and the importance of determining how to optimize the technical operation of a 
process (in addition to economic optimization). 

All the words in Cluster 4 include the word “heuristic” even though the three terms are 
not related in a deeper, more fundamental way. It is likely that this is an example of the 
use of surface characteristics to group terms. We note that Cluster 4 remains intact in the 
posttest and in fact clusters even closer (the three terms are plotted as the exact same 
point). Thus the use of surface characteristics to cluster these terms seems to become 
stronger at the end of the semester even though the course included several lengthy 
discussions about the utility of heuristics to perform rapid design calculations and check 
detailed computer simulation results. This may indicate that even at the end of the 
course, students do not have a deep understanding of these terms. However, since these 
terms move closer to terms relating to process analysis, students may have a better 
understanding of how design, equipment, and operating heuristics can be utilized in 
process analysis and design. Thus the MDS results may indicate that students have some 
understanding of how to use the heuristics, but probably do not yet fully understand 
where the heuristics come from and their limitations. 

Two terms, HAZOP analysis and risk analysis, should have been plotted closely together 
but were not in the pretest or posttest. In the posttest the term HAZOP analysis moves 
into Cluster 3 but is still not located near risk analysis. These results suggest that 
students do not yet have sufficient understanding of risk in the design and operation of 
hazardous chemical processes, so this topic will be emphasized more thoroughly in future 
offerings of the process design course. 

Both pretest and posttest results also suggest that the term “life-cycle analysis” may be 
unclear in the students’ minds. Although an important concept in terms of analyzing a 
chemical product from “cradle to grave,” the design course does not devote much time to 
the life cycle of chemical species, and not surprisingly, the design students have little 
familiarity with the concept. 

The term troubleshooting remains unclustered with any other terms in both the pretest 
and the posttest. Although several weeks of class time were devoted to completing 
process troubleshooting case studies, this result may indicate that students do not 
understand how this concept relates to process design and analysis. 

Educational implications 

In this study we have used MDS as a classroom assessment technique (Angelo & Cross, 
1993), a way of quickly gathering information about student progress which then can be 
used as a feedback mechanism to alter instruction and improve student understanding. 

We hope that further development of the on-line MDS web page would make it easier for 
instructors to use MDS as an assessment tool in their courses. 

The results of the MDS cluster analysis have been used to help indicate where students 
have accurate knowledge of engineering design (for example, when discussing the 
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economic analysis of a design project) and where their knowledge may be incomplete 
(for example, when discussing life-cycle analysis, risk analysis, and troubleshooting). 
This information has been used to guide the development and implementation of a new 
module on risk assessment and HAZOP analysis in the process design course. 

Based on the results of this study, we also believe MDS can be utilized as a method for 
identifying misconceptions that students bring to the course (i.e. that different types of 
heuristics should be lumped into the same conceptual cluster instead of considering 
different types of heuristics as distinct tools for use on different design tasks). This 
information is now being used by the design course instructors to make more explicit the 
similarities, differences, and limitations of the types of design heuristics used in the 
course. New exercises have been developed to help students discover the origin and use 
of each type of heuristic. 

We plan to use the MDS tool in future offerings of the design course to monitor the 
impact of the course changes mentioned above on student understanding of chemical 
engineering design strategies and techniques. 
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